{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Overview\n",
    "\n",
    "- **Expectation Used:** [expect_column_values_confidence_for_data_label_to_be_greater_than_or_equal_to_threshold](https://github.com/great-expectations/great_expectations/blob/develop/contrib/capitalone_dataprofiler_expectations/capitalone_dataprofiler_expectations/expectations/expect_column_values_confidence_for_data_label_to_be_greater_than_or_equal_to_threshold.py)\n",
    "\n",
    "- **Expectation Description:** This expectation will check every record in the user specified column to determine if any given value is detected under the user-specified label with a confidence greater than or equal to the user-specified threshold. The actual confidence level is generated by the Data Profiler's Data Labeler when it processes each record.\n",
    "\n",
    "- **Use Case:** If a user has sensitive data, such as SSNs, that they would expect to be detected as SSN label at a confidence level greater than or equal to a certain threshold, then they could use this expectation to identify and records which fall below this threshold.\n",
    "\n",
    "- **Example Details:** In this example, lets assume a data owner has a dataset that holds salary information about individuals in the data science field. The dataset has uuids which uniquely identifies each record. Let's also assume that the uuid column is a join variable representing a unique individual. The data owner may want more insight into the Data Labeler's confidence levels on validating these uuid records as true `UUID` values. This is to ensure joins are conducted on high quality key columns."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Imports"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Great expectations imports\n",
    "import great_expectations as ge\n",
    "from capitalone_dataprofiler_expectations.expectations. \\\n",
    "    expect_column_values_confidence_for_data_label_to_be_greater_than_or_equal_to_threshold \\\n",
    "    import ExpectColumnValuesConfidenceForDataLabelToBeGreaterThanOrEqualToThreshold\n",
    "from great_expectations.self_check.util import build_pandas_validator_with_data\n",
    "\n",
    "# Data Profiler imports\n",
    "import dataprofiler as dp"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Setup\n",
    "Below we are going to import a dataset from the Data Profile testing suite. This csv holds information on the salaries of individuals in the data science field from all over the world."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "context = ge.get_context()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data_path = \"../../dataprofiler/tests/data/csv/ds_salaries.csv\"\n",
    "data = dp.Data(data_path).data\n",
    "data"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Running the Exception\n",
    "We build the validator by passing in the dataframe that has been built above. Then we will use the exception below to check that records from the `uuid` column detected by the Data Labeler as `UUID` labels with a confidence greater than or equal to 0.9. Any records that fall below this threshold will trigger a violation in the expectation report indicating to the data owner which uuids do not satisfy the expectation."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "validator = build_pandas_validator_with_data(data)\n",
    "results = validator.expect_column_values_confidence_for_data_label_to_be_greater_than_or_equal_to_threshold(\n",
    "    column='uuid',\n",
    "    data_label='UUID',\n",
    "    threshold=.90\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Results\n",
    "From the output below, the data owner can see that the expectation has pass successfully. This indicates that all records in the `uuid` column are true `UUID` values with a confidence level greater than or equal to 0.9. Therefore, the ETL pipeline is protected from unsafe joins that could cause data issues further in the process."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "results"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
